Object-Based Image Retrieval Beyond Visual Appearances

The performance of object-based image retrieval systems remains unsatisfactory, as it relies highly on visual similarity and regularity among images of same semantic class. In order to retrieve images beyond their visual appearances, we propose a novel image presentation, i.e. bag of visual synset. A visual synset is defined as a probabilistic relevance-consistent cluster of visual words (quantized vectors of region descriptors such as SIFT), in which the member visual words w induce similar semantic inference P(c|w) towards the image class c. The visual synset can be obtained by finding an optimal distributional clustering of visual words, based on Information Bottleneck principle. The testing on Caltech-256 datasets shows that by fusing the visual words in a relevance consistent way, the visual synset can partially bridge visual differences of images of same class and deliver satisfactory retrieval of relevant images with different visual appearances.